Logistic regression is the go-to linear classification algorithm for two-class problems. It is easy to implement, easy to understand and gets great results on a wide variety of problems, even when the expectations the method has for your data are violated.
Logistic regression is named for the function used at the core of the method, the logistic function.
The logistic function, also called the Sigmoid function was developed by statisticians to describe properties of population growth in ecology, rising quickly and maxing out at the carrying capacity of the environment. It’s an S-shaped curve that can take any real-valued number and map it into a value between 0 and 1, but never exactly at those limits.
$e$ is the base of the natural logarithms and $x$ is value that you want to transform via the logistic function.
In [1]:
import numpy as np
import matplotlib.pyplot as plt
import seaborn
%matplotlib inline
In [2]:
x = np.linspace(-6, 6, num = 1000)
plt.figure(figsize = (12,8))
plt.plot(x, 1 / (1 + np.exp(-x))); # Sigmoid Function
plt.title("Sigmoid Function");
The logistic regression equation has a very simiar representation like linear regression. The difference is that the output value being modelled is binary in nature.
$$\hat{y}=\frac{e^{\beta_0+\beta_1x_1}}{1+\beta_0+\beta_1x_1}$$or
$$\hat{y}=\frac{1.0}{1.0+e^{-\beta_0-\beta_1x_1}}$$$\beta_0$ is the intecept term
$\beta_1$ is the coefficient for $x_1$
$\hat{y}$ is the predicted output with real value between 0 and 1. To convert this to binary output of 0 or 1, this would either need to be rounded to an integer value or a cutoff point be provided to specify the class segregation point.
In [3]:
tmp = [0, 0.4, 0.6, 0.8, 1.0]
In [4]:
tmp
Out[4]:
In [5]:
np.round(tmp)
Out[5]:
In [6]:
np.array(tmp) > 0.7
Out[6]:
$\beta_0$ is the intecept term
$\beta_1$ is the coefficient for $x_i$
$\hat{y}$ is the predicted output with real value between 0 and 1. To convert this to binary output of 0 or 1, this would either need to be rounded to an integer value or a cutoff point be provided to specify the class segregation point.
In [7]:
dataset = [[-2.0011, 0],
[-1.4654, 0],
[0.0965, 0],
[1.3881, 0],
[3.0641, 0],
[7.6275, 1],
[5.3324, 1],
[6.9225, 1],
[8.6754, 1],
[7.6737, 1]]
Let's say you have been provided with the coefficient
In [8]:
coef = [-0.806605464, 0.2573316]
In [9]:
for row in dataset:
yhat = 1.0 / (1.0 + np.exp(- coef[0] - coef[1] * row[0]))
print("yhat {0:.4f}, yhat {1}".format(yhat, round(yhat)))
The coefficients (Beta values b) of the logistic regression algorithm must be estimated from your training data. This is done using maximum-likelihood estimation.
Maximum-likelihood estimation is a common learning algorithm used by a variety of machine learning algorithms, although it does make assumptions about the distribution of your data (more on this when we talk about preparing your data).
The best coefficients would result in a model that would predict a value very close to 1 (e.g. male) for the default class and a value very close to 0 (e.g. female) for the other class. The intuition for maximum-likelihood for logistic regression is that a search procedure seeks values for the coefficients (Beta values) that minimize the error in the probabilities predicted by the model to those in the data (e.g. probability of 1 if the data is the primary class).
We are not going to go into the math of maximum likelihood. It is enough to say that a minimization algorithm is used to optimize the best values for the coefficients for your training data. This is often implemented in practice using efficient numerical optimization algorithm (like the Quasi-newton method).
When you are learning logistic, you can implement it yourself from scratch using the much simpler gradient descent algorithm.
In [10]:
from sklearn.linear_model import LogisticRegression
In [11]:
dataset
Out[11]:
In [13]:
X = np.array(dataset)[:, 0:1]
y = np.array(dataset)[:, 1]
In [14]:
X
Out[14]:
In [15]:
y
Out[15]:
In [22]:
clf_LR = LogisticRegression(C=1.0, penalty='l2', tol=0.0001)
In [23]:
clf_LR.fit(X,y)
Out[23]:
In [24]:
clf_LR.predict(X)
Out[24]:
In [25]:
clf_LR.predict_proba(X)
Out[25]: